Predictive Control of Speededness in Adaptive Testing (RR 07-02) (PDF)

An adaptive testing method is presented that controls the speededness of the test using updated predictions of the response times of the test taker on the candidate items in the pool. Two different types of updates were investigated: (1) posterior predictions given the actual response times on the items already administered and (2) posterior predictions that use the responses on these items as an additional source of information. In a simulation study with an adaptive test from the Armed Services Vocational Aptitude Battery (ASVAB), the effectiveness of the methods in removing differential speededness from the test was evaluated. Predictive Control of Speededness in Adaptive Testing Test items differ in the amount of time they require to produce a response to them. These differences are due, for instance, to the amount of information decoding and processing they involve, the complexity of their problem-solving process, and the numbers of concepts or relations that have to be retrieved from memory. It is not unusual to find differences between the time intensities of the items in a pool for an adaptive test equal to a factor of five or larger (van der Linden & Guo, 2006). It is important to distinguish the time intensity of test items from the speed at which the test takers respond to them. Within a certain range, test takers have a choice of the speed at which they work. When making such a choice, they have to deal with what is known in reaction-time research in psychology as a speed-accuracy trade-off (i.e., Luce, 1986, sect. 2.2.7). The trade-off implies that if a higher speed is chosen, the accuracy of problem-solving will go down. Accuracy in problem-solving is defined as the probability of making an error. In test theory, the probability of success on a test item is typically modeled as a function of an ability parameter. Thus, we can also refer to the trade-off as a speed-ability trade-off. The trade-off implies that the test taker’s ability level during the test depends on his or her choice of speed. We will denote the specific ability level that is the result of this choice as the effective ability of the test taker. In adaptive testing, each test taker gets a different selection of items. Because these selections differ in time intensity, some of the test takers may have to work under high pressure whereas others have ample time to complete the test. The same phenomenon of differential speededness may occur in multistage testing or in other multiple-form testing formats, such as linear-on-the-fly testing. But we expect the effects of differential speededness to be both more serious and systematic for testing formats with adaptive item selection. Typically, more difficult items tend to be more time-intensive. Adaptation then gives the time-intensive items to the more able test takers. The presence of a speed-ability trade-off implies that the administration of items of unequal time intensity is not inconsequential. If the time limit is economically chosen, test takers who get the more time-intensive items are forced to make a choice between increasing their speed and accepting a lower effective ability or maintaining it and guessing on the items that are not reached. In either case, the effect is a 1